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S3 ; Abstract 

We study a class of backward stochastic differential equations (BSDEs) driven by a ran- 
£T) ■ dom measure or, equivalently, by a marked point process. Under appropriate assumptions 

CN ! we prove well-posedness and continuous dependence of the solution on the data. We next ad- 

dress optimal control problems for point processes of general non-markovian type and show 
m^j \ that BSDEs can be used to prove existence of an optimal control and to represent the value 

function. Finally we introduce a Hamilton- Jacobi-Bellman equation, also stochastic and of 
backward type, for this class of control problems: when the state space is finite or countable 
we show that it admits a unique solution which identifies the (random) value function and 
can be represented by means of the BSDEs introduced above. 
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i . 1 Introduction 

in ■ 

The purpose of this paper is to study a class of backward stochastic differential equations (BSDEs 
for short) and apply these results to solve optimal control problems for marked point processes. 
Under appropriate assumptions, an associated Hamilton-Jacobi-Bellman equation of stochastic 
type is also introduced and solved in this non-markovian framework. 

General nonlinear BSDEs driven by the Wiener process were first solved in [20] . Since then, 
many generalizations have been considered where the Wiener process was replaced by more 
general processes. Among the earliest results we mention in particular [13], [14], to which some 
of our results are inspired, and we refer e.g. to [7] for a recent result and for indications on the 
existing bibliography. 

We address a class of BSDEs driven by a random measure, naturally associated to a marked 
point process. There exists a large literature on this class of processes, and in particular to 
the corresponding optimal control problems: we only mention the classical treatise [6] and the 
recent book [5] as general references. In spite of that, there are relatively few results on their 
connections with BSDEs. In the general formulation of a BSDE driven by a random measure, 
one of the unknonwn processes (the one associated with the martingale part, or Z-process) is 
in fact a random field. This kind of equations has been introduced in [24] . and has been later 
considered in [2], [23J in the markovian case, where the associated (nonlocal) partial differential 
equation and related non-linear expectations have been studied. 

In these papers the BSDE contains a diffusive part and a jump part, but the latter is 
only considered in the case of a Poisson random measure. In order to give a probabilistic 
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representation of solutions to quasi-variational inequalities in the theory of stochastic impulse 
control, in [18] a more difficult problem involving also constraints on the jump part is formulated 
and solved, but still in the Poisson case and in a markovian framework. 

To our knowledge, the only general result beyond the Poisson case is the paper [26]. Here, 
under conditions of Lipschitz type on the coefficients and assuming the validity of appropriate 
martingale representation theorems, a general BSDE driven by a diffusive and a jump part is 
considered and well-posedness results and a comparison theorem are proven. However, it seems 
that in this paper the formulation of the BSDE was not chosen in view of applications to optimal 
control problems. Indeed, in contrast to [21] or [2J, the generator of the BSDE depends on the 
Z-process in a specific way (namely as an integral of a Nemytskii operator) that is generally not 
valid for the hamiltonian function of optimal control problems (compare for instance formula 
(|1.3p below) and therefore prevents direct applications to these problems. 

In our paper we consider a BSDE driven by a random measure, without diffusion part, on a 
finite time interval, of the following form: 

Y t + [ T [ Z s (y)q(dsdy)=^ + f f s (Y s ,Z s (-))dA s , t E [0,T], (1.1) 
Jt JK Jt 

where the generator / and the final condition £ are given. 

Here the basic probabilistic datum is a marked point process (T n ,£ n ) where (T n ) is an 
increasing sequence of random times and (£ n ) a sequence of random variables in the state 
(or mark) space K. The corresponding random counting measure is p(dtdy) = X) n <£(T n ,£ n )> 
where 5 denotes the Dirac measure. We denote (At) the compensator of the counting process 
(p([0, £] x K)) and by <j>t(dy)dAt the (random) compensator of p. Finally, the compensated 
measure q(dtdy) = p(dtdy) — 4>t(dy)dA t occurs in equation (jl.ip . The unknown process is 
a pair (Y t ,Z t (-)), where Y is a real progressive process and {Z t (y), t G [0, T], y G K} is a 
predictable random field. 

The random measure p is fairly general, the only restriction being non explosion (i.e. T n — > 
oo) and the requirement that (At) has continuous trajectories. We allow the space K to be of 
general type, for instance a Lusin space. Therefore our results can also be directly applied to 
marked point processes with discrete state space. We mention at this point that the specific case 
of finite or countable Markov chains has been studied in [8], [9], see also |10j for generalizations. 

The basic hypothesis on the generator / is a Lipschitz condition requiring that for some 
constants L > 0, L 1 > 0, 



\ft(u,,r,z(-))-f t (u,r',z'(-))\ < L'\r - r'\ + L N\z(y) - z'(y)\ 2 M",dy) 
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for all (cj, t), for r,r' G R, and z,z' in appropriate function spaces (depending on (oo,t)): see 
below for precise statements. We note that the generator of the BSDE can depend on the 
unknown Z-process in a general functional way: this is required in the applications to optimal 
control problems that follow, and it is shown that our assumptions can be effectively verified in 
a number of cases. In order to solve the equation, beside measurability assumptions, we require 
the summability condition 

fT 



E f e' 3At |/ i (0,0)| 2 ^ + E[ e ^|e| 2 ]<oo, 
J o 



to hold for some f3 > I? + 21J . Note that in the Poisson case mentioned above we have a deter- 
ministic compensator 4>t(dy) dA t = n(dy) dt for some fixed measure tt on K and the summability 
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condition reduces to a simpler form, not involving exponentials of stochastic processes. We prove 
existence, uniqueness, a priori estimates and continuous dependence upon the data for the so- 
lution to the BSDE. 

The results described so far are presented in section [31 after an introductory section devoted 
to notation and preliminaries. 

In section H] we formulate a class of optimal control problems for marked point processes, 
following a classical approach exposed for instance in JB]. For every fixed (t, x) £ [0, T] x K, the 
cost to be minimized and the corresponding value function are 



J t (x,u{-))=E: 



T 



J l s (X t s > x ,u s )dA s +g(X t /) 



v(t, x) = ess inf Jt(x, u(-)), 



where E^' denotes the conditional expectation with respect to a new probability P u , depending 
on a control process (ut) and defined by means of an absolutely continuous change of measure: 
the choice of the control process modifies the compensator of the random measure under F u 
making it equal to rt(y,ut)4>t(dy) dAt for some given function r. To this control problem we 
associate the BSDE 

Y**+ [ T [ Z^(y)q(drdy) = g(X t /)+ F f (r, X^ , (•)) dA r , s € [t,T\. (1.2) 

Js JK Js 

where {Xl ,x ) is a family of marked point processes, each starting from x at time i, and the 
generator contains the hamiltonian function 

f(u, t, x, z(-)) = inf <^ l t (u, x, u) + / z(y) (r t (w, y, u) - 1) 4> t (uj, dy) \ . (1.3) 
u>£U I J K J 

Assuming that the infimum is in fact a minimum, admitting a suitable selector, together with a 
summability condition of the form 

Eexp(/M T ) +E[\g(X^ x )\ 2 e pAT ] < oo 

for a sufficiently large value of /?, we prove that the optimal control problem has a solution, and 
that the value function and the optimal control can be represented by means of the solution to 
the BSDE. 

We note that optimal control of point processes is a classical topic in stochastic analysis, 
and the first main contributions date back several decades: we refer the reader for instance to 
the corresponding chapters of the treatises [6] and [15]. The markovian case has been further 
investigated in depth, even for more general classes of processes, see e.g. [12]. The results 
we present in this paper are an attempt toward an alternative systematic approach, based on 
BSDEs. We hope this may lead to useful results in the future, for instance in connection with 
computational issues and a better understanding of the nonmarkovian situation. Although this 
approach is analogous to the diffusive case, it seems that it is pursued here for the first time in 
the case of marked point processes. In particular it differs from the control-theoretic applications 
addressed in [21], devoted to a version of the stochastic maximum principle. 

Finally, in section [5l we introduce the following Hamilton- Jacobi-Bellman equation (HJB for 
short) associated to the optimal control problem described above: 

v(t,x)+ / / V(s,x,y)q(dsdy) 



T 



= 9{x) + J J (v(s,y) -v(a,x) + V(a,y,y) -V(s t x t yf) <j) s (dy)dA s (1.4) 
+ J f(s,x,v(s,-)-v(s,x)+V(s,;-))dA s , te[Q,T],xeK, 
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where / be the hamiltonian function defined in (|1.3p . The solution is a pair of random fields 
{v(t, x), V(t, x, y) : t £ [0, T], x,y £ K}, and in this non-markovian framework the HJB equa- 
tion is stochastic and of backward type, driven by the same random measure as before. Thus, 
the previous results are applied to prove its well-posedness. For technical reasons, however, 
we limit ourselves to the case where the state space K is at most countable: although this 
is a considerable restriction with respect to the previous results, it allows to treat important 
classes of control problems, for instance those related to queuing systems. Under appropriate 
assumptions, similar to those outlined above, we prove that the HJB equation is well-posed and 
that v (t, x) coincide with the (stochastic) value function of the optimal control problem and it 
can be represented by means of the associated BSDE. A backward stochastic HJB equation has 
been first introduced in [21] in the diffusive case, where the corresponding theory is still not 
complete due to greater technical difficulties. It is an interesting fact that the parallel case of 
jump processes can be treated using BSDEs and fairly complete results can be given, at least 
under the restriction mentioned above: this is perhaps due to the different nature of the control 
problem (here the laws of the controlled processes are obtained via an absolutely continuous 
change of measure, in contrast to [H]). We borrow some ideas from [21], in particular the use 
of a formula of Ito-Kunita type proven below, that suggested the unusual form of (jl.4p . We are 
not aware of any previous result on backward HJB equations in a non-diffusive context. 

The results of this paper admit several variants and generalizations: some of them are not 
included here for reasons of brevity and some are presently in preparation. For instance, the 
BSDE approach to optimal control of Markov jump processes deserves a specific treatment; 
moreover, BSDEs driven by random measures can be studied without Lipschitz assumptions on 
the generator, along the lines of the many results available in the diffusive case, or extensions 
to the case of vector-valued process Y or of random time interval can be considered. 

2 Notations, preliminaries and basic assumptions 

In this section we are going to recall basic notions on marked points processes, random measures 
and corresponding stochastic integrals, that will be constantly used in the rest of the paper. We 
also formulate several assumptions that will remain in force throughout. 

2.1 Marked point processes 

Let (fi, J-, P) be a complete probability space and (K, fC) a measurable space. Assume we have a 
sequence (T n ,£ n ) n >i of random variables, T n taking values in [0, oo] and £ n in K. We set Tq = 
and we assume, P-a.s., 

T n < oo T n < T n+1 , ra > 0. 

We call (T n ) a point process and (T n ,£ n ) a marked point process. K is called the mark space, 
or state space. 

In this paper we will always assume that (T n ) is nonexplosive, i.e. T n — > oo P-a.s. 
For every A £ K, we define the counting processes 

N t (A) = £ l T „<t W, t > 

n>l 

and we set N t = N t (K). We define the filtration generated by the counting processes by first 
introducing the er-algebras 

jt-o = a(N s (A) : s £ [Q,t], A £ XT), t > 0, 
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and setting 

T t = a(J*,M), t>0, 

where J\f denotes the family of P- null sets in T . It turns out that {Tt)t>o is right-continuous and 
therefore satisfies the usual conditions. In the following all measur ability concepts for stochastic 
processes (e.g. adaptedness, predictability) will refer to the filtration {Ft)t>o- The predictable a- 
algebra (respectively, the progressive cr-algebra) on £2 x [0, oo) will be denoted by V (respectively, 
by Prog). The same symbols will also denote the restriction tofix [0, T] for some T > 0. 

It is known that there exists an increasing, right-continuous predictable process A satisfying 
Aq = and 

roc fco 

E / /// >I.X, E / H t dA t 
Jo Jo 

for every nonnegative predictable process H. The above stochastic integrals are defined for 
P-almost every oo as ordinary (Stieltjes) integrals. A is called the compensator, or the dual 
predictable projection, of N. In the following we will always make the basic assumption that 
P-a.s. 

A has continuous trajectories (2-1) 

which are in particular finite- valued. 

We finally fix £o G K (deterministic) and we define 

X t = J2^[T n ,T n+1 )(t), t>0. (2.2) 

n>0 

We do not assume that P(£ n 7^ fn+l) = 1- Therefore in general trajectories of (T n ,£ n ) n >o 
cannot be reconstructed from trajectories of (X t )t>o and the filtration {Ft)t>o is not the natural 
completed filtration of (X t )t>o- 

2.2 Random measures and their compensators 

For weSlwe define a measure on ((0, oo) x K, £>((0, oo)) ® fC) setting 

P (oj, C) = J2 l(T„H,€nH)ec C G B((0, oo)) ® K, 

n>l 

where B(A) denotes the Borel a-algebra of any topological space A. p is called a random measure 
since oj i->- p(oj, C) is ^-measurable for fixed C. We also use the notation p(u, dtdy) or p(dtdy). 
Notice that p((0,t] x A) = N t (A) for t > 0,A G /C. 

Under mild assumptions on K it can be proved that there exists a function cfit(uj,A) such 
that 

1. for every u G $7, i G [0, oo), the mapping A h-» ^4) is a probability measure on (if, /C); 

2. for every i £ /C, the process (u),t) t-t (f>t(u),A) is predictable; 

3. for every nonnegative Ht(oJ,y), V <S> /C-measurable, we have 

roc poo 

E / H t (y) p{dtdy) = E / H t (y) <t> t {dy) dA t . 
Jo Jo 

For instance, this holds if (K, /C) is a Lusin space with its Borel cr-algebra (see [17] Section 
2), but since the Lusin property will not play any further role below, in the following we will 
simply assume the existence of 4>t(dy) satisfying 1-2-3 above. 

The random measure 4>t(oj,dy) dAt(u) will be denoted p(u,dtdy), or simply p(dtdy), and 
will be called the compensator, or the dual predictable projection, of p. 
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2.3 Stochastic integrals 

Fix T > 0, and let H t (co, y) be a V ® /C-measurable real function satisfying 



/ / |if t (y)| Mdy)dA t < oo, P-o.s. 

JO J if 



'0 JK 

Then the following stochastic integral can be defined 

f [ H s (y)q(dsdy):= [ f H s (y) p(ds dy) - I [ H s (y) <f> s (dy) dA s , t G [0, T], (2.3) 
Jo Jk Jo Jk Jo Jk 

as the difference of ordinary integrals with respect to p and p. Here and in the following the 
symbol is to be understood as an integral over the interval (a, b] . We shorten this identity 
writing q(dt dy) = p(dt dy) — p(dt dy) = p(dt dy) — <f>t(dy) dA t . Note that 

f* f H s {y)p{dsdy)= £ H Tn (U) 
Jo Jk n>l,T n <t 

is always well defined since we are assuming that T n — > oo P-a.s. 

For r > 1 we define C r,0 (p) as the space of V /C-measurable real functions Ht(u,y) such 
that 

E/ [ \H t (y)\ r p(dtdy)=E [ [ \H t (y)\ r </> t (dy) dA t < oo 
jo Jk jo Jif 

(the equality of the integrals follows from the definition of <pt(dy)). Given an element H of 
C 1,0 (p), the stochastic integral (j2.3|) turns out to be a a finite variation martingale. 

The key result used in the construction of a solution to the BSDE f)3. 1 1) is the integral 
representation theorem of marked point process martingales (see e.g. [H],[32]), which is a 
counterpart of the well known representation result for Brownian martingales (see e.g. |22| 
Ch V.3 or [TS] Thm 12.33). Recall that {Ft) is the filtration generated by the jump process, 
augmented in the usual way. 

Theorem 2.1 Let M be a cadlag (Ft) -martingale on [0, T\. Then we have 

M t = M + [ [ H s (y) q(ds dy), t G [0, T], 
Jo Jk 

for some process H £ C l,0 (p). 



2.4 A family of marked point processes. 

In the following, in order to use dynamic programming arguments, it will be useful to introduce 
a family of processes instead of the single process X, each starting at a different time from 
different points. 

Let (T n ,£ n ) be the marked point process introduced in section 12.11 We fix t > and we 
introduce counting processes relative to the time interval [t, oo) setting 

N l s (A) = ^2 1 t<Tn<sM n eA, s e [t,oo), AeJC, 

n>l 

and Ng = Ng(K). Then N*(A) = p l ((t, s] x A) for s > t, A £ /C, where the random measure p t is 
the restriction of p to (t, oo) x K. With these definitions it is easily verified that the compensator 
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of p t (respectively, iV 4 ) is the restriction of 4> s (dy) dA s (respectively, A) to [t, oo)xK (respectively, 
[t,oo)). 

Now we fix t > and x G K. Noting that Nt is the number of jump times T n in the interval 
[0,t], so that T\r t < t < Tjv t +i, we define 

=ll [t , TjV(+l) (s)+ Cnl[T„,T n+1 )( S )' S€ [*,(»). 

n>iV t +l 

In particular, recalling the definition of the process X, previously defined by formula (12. 2\i and 
starting at point £o £ K, we observe that X = X°^°. 

For arbitrary t, x we also have Xt' x = X s for s > T/v t +i and, finally, for < u < t < s and 
x £ K the identity xj' ' = Xs"' 1 is easy to verify. 

3 The backward equation 

From now on, we fix a deterministic terminal time T > 0. 

For given uj £ Q and t £ [0, T], we denote C r (K, fC, 4>t(u, cZy)) the usual space of /C-measurable 
maps z : K — > R such that J x |z(y)| r ^t(o;, dy) < 00 (below we will only use r = or 1). 

Next we introduce several classes of stochastic processes, depending on a parameter /? > 0. 

• £pJ: ofl (ft x [0,T]) denotes the set of real progressive processes Y such that 

\Y\l :=E [ e pAt \Y t \ 2 dA t < 00. 
Jo 

• C 2, ^(p) denotes the set of mappings Z :VL x [0, T] x K — > R which are T 3 (X 1 /C-measurable 
and such that 



Z||J := E / / e^|Z t (y)|V f W^ < 00. 

io JK 



We say that Y, Y' £ C 2 f rog {^ x [0,T]) (respectively, Z, Z' G £ 2l/3 (p)) are equivalent if they 
coincide almost everywhere with respect to the measure dAt(uj)¥(du) (respectively, the measure 
(j)t(uj,dy)dAt(u))P(duj)) and this happens if and only if |Y— Y'^ = (respectively, \\Z—Z'\\p = 0). 
We denote L^ og (U x [0, T]) (respectively, L 2,l3 (p)) the corresponding set of equivalence classes, 
endowed with the norm | • \p (respectively, || • \\p). L 2 p^ og (Q x [0, T]) and L 2 ^(ji) are Hilbert 
spaces, isomorphic to L 2 ^(9, x [0, T] , Prog, e^ A * ^ dA t (cj) P(dw)) and L 2 'P{VL x [0,T] x K,V ® 
K, ePM") (f) t (oj, dy) dA t (u) P(dw)) respectively. 

Finally we introduce the Hilbert space K 13 = L 2 / rog {tt x [0,T]) x L 2 'P(p), endowed with the 
norm\\(Y,Z)\\ 2 :=\Y\ 2 +\\Z\\ 2 . 

In the following we will consider the backward stochastic differential equation: P-a.s., 

Y t + [ T [ Z s {y)q{dsdy)=i+ F f s (Y s ,Z s (-))dA s , t £ [0,T], (3.1) 
Jt JK Jt 

where the generator / and the final condition £ are given and and we look for unknown processes 
(Y,Z) G K$. 

Let us consider the following assumptions on the data / and £. 
Hypothesis 3.1 1. The final condition £ : Q. — > R is J 7 ? -measurable and Ee^ Ar \^\ 2 < 00. 
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2. For every uj G Q, t G [0,T], r£R, a mapping f t (oj,r, •) : C 2 {K,K,,4>t{^,dy)) — ► R is given, 
satisfying the following assumptions: 

(i) for every Z G C 2 '^{p) the mapping 

(u,t,r)^f t (u,r,Z t (u,-)) (3-2) 

is Prog (g) B(M) -measurable; 

(ii) there exists L > 0, L' > suc/i i/iai /or ewen/ uj G 0, i G [0,T], r, r' G R, z 1 G 
£ 2 (K,K,,(f)t{u,dy)) we have 

\f t (uj,r,z(-))- f t (u,r' ',/(•))! < L'\r-r'\ + L^J \z(y) - z'{y)\ 2 <t> t {u,dy)\ ' ; (3.3) 
(in) We have 

l/ e ^|/ t (0,0)| 2 cL4* <oo. (3.4) 
J o 



Remark 3.2 1. The slightly involved measur ability condition on the generator seems un- 
avoidable, since the mapping ft(uj,r,-) has a domain which depends on (uj,t). However, 
in the following section, we will see how it can be effectively verified in connection with 
optimal control problems. 

Note that if Z G C 2, ^{p) then Z t (uj, •) belongs to C 2 (K,)C,4>t(uj,dy)) except possibly on a 
predictable set of points (uj,t) of measure zero with respect to dAt(uj)¥(du), so that the 
requirement on the measurability of the map (|3,2p is meaningful. 

2. We note the inclusion 

C 2,f5 {p) c C l,0 {p y (3 5) 

Indeed if Z G C 2 '"(p) then the inequality 

r \Z t (y)\ Mdy)dA t < ( [ T [ \Z t {y)\ 2 Mdy)e? At dA t ) ' ( f e~ pAt dA t ) ' 

JK \J0 JK / \J0 / 

and the fact that ^ e~ fiM dA t = ^(l - e~^ AT ) < (3' 1 imply that Z G C lfi (p). 

It follows from (|3.5p that the martingale M t = f Q f K Z s (y) q(ds dy) is well defined for 
Z G C 2) P{p) and has cadlag trajectories P-a.s. It is easily checked that M only depends on 
the equivalence class of Z as defined above. 

Lemma 3.3 Suppose that f : £1 x [0, T] — > R is progressive, £ : $7 — > R is Ft -measurable and 

pT 

Ee 0AT \£\ 2 +E e^ As \f s \ 2 dA s < oo 

JO 

/or some j3 > 0. Then there exists a unique pair (Y, Z) in ¥*P solution to the BSDE 

Y t + [ T [ Z s (y) q(ds dy)=i+ C f s dA s . (3.6) 
Jt JK Jt 

Moreover the following identity holds: 

rT 



Ee^ At \Y t \ 2 + /3E [ e^\Y s \ 2 dA s + E [ ! eP A *\Z s (y)\ 2 <t> s {dy)dA s 

Jt Jt JK 
rT 

= Ee' 3Ar |£| 2 + 2E^ e^Ys f s dA s , t £ [0,T], 



(3.7) 
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and and there exist two constants c\{j3) = 4(1 + ^) and C2(/3) = ^(1 + jj) such that 

E f e^\Y s \ 2 dA s +K f T [ e^\Z s (y)\ 2 Mdy)dA s < Cl (f3)Ee^\^\ 2 +c 2 ((3) F e^|/ s | 2 dA s . 
Jo Jo Jk Jo 

(3.8) 



Proof. Uniqueness follows immediately using the linearity of (|3.6p and taking the conditional 
expectation given Tt- 

Assuming that (Y, Z) 6 WL 13 is a solution, we now prove the identity (|3.7p . From the Ito 
formula applied to e /3j4 *|y"t| 2 it follows that 

d{e pAt \Y t \ 2 )= ^ At \Y t \ 2 dA t + 2e pAt Y t -dY t + e? At \ AY t \ 2 . 

So integrating on [t, T] and recalling that A is continuous, 



e^\Y t \ 2 = -[ Pe^\Y s \ 2 dA s -2 f e^Y s _ [ Z s (y)q(ds dy) - Ve^|AY s | 2 
Jt Jt Jk t<s < T 

+e^|£| 2 + 2 J* e? A °Y s f s dA s . (3.9) 

The integral process J*q e^ s Ys_ Z s (y)q(ds dy) is a martingale, because the integrand process 
e@ s Y s -.Z s (y) is in /^(p): in fact from the Young inequality we get 



E / / e^\Y 8 _\\Z a {y)\<t> a {dy)dA 
lo Jk 

< 

2 

Moreover we have 



^E [ T e^\Y s .\ 2 dA s + ^ F [ ^|Z s ( y )|%(dy)d^<+cx5. 

2 JO ^ JO JA" 



V e^\AY s \ 2 = f ! e^\Z s {y)\ 2 p{dsdy) 
Jo Jk 

e^\Z s (y)\ 2 q(dsdy) + f [ e^\Z s (y)\ 2 Mdy)dA s , 

Jo Jk 



0<s<t 



io Jk 

where the stochastic integral with respect to q is a martingale. Taking the expectation in (|3.15p 
we obtain (13. 7p . 

We now pass to the proof of existence of the required solution. We start from the inequality 

£ \f s \ dA s = £ e-^e^\f s \ dA s < (J* e^dA^ " e^\f s \ 2 dA s ^ ^ . 

Since f3 e-^ Aa dA s = e~^ At - e~^ AT < e _/3At we arrive at 

\f.\dA.) <—J t e^\f s \ 2 dA s . (3.10) 

This implies in particular that \f s \ dA s is square summable. The solution (Y,Z) is defined 
by considering a cadlag version of the martingale Mt = E^*^ + f s dA s }. By the martingale 
representation Theorem 12. 1\ there exists a process Z £ C 1,0 (p) such that 

M t = M + [ [ Z s {y)q(dyds), t € [0,T\. 
Jo Jk 
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Y t = M t - f f s (U s , V s ) dA s , t E [0, T]. 
Jo 



Define the process Y by 



Noting that Yt = £, we easily deduce that the equation (|3.6p is satisfied. 

It remains to show that (Y, Z) E K^. Taking the conditional expectation, it follows from 
fl3H that Y t = E^' [£ + / s <L4 S ] so that, using (flTTOD . we obtain 

2 



e^|y t | 2 < 2e /3At |E J "^| 2 + 2e /3At 



/ s eL4< 



e ^|e| 2 + I / e^\f s \ 2 dA 
P Jo 



(3.11) 



Denoting by mt the right-hand side of (|3.1ip . we see that m is a martingale by the assumptions 
of the lemma. In particular, for every stopping time S with values in [0,T], we have 

Ee pAs \Y s \ 2 < Em s = Em T < oo (3.12) 

by the optional stopping theorem. Next we define the increasing sequence of stopping times 

S n = inf{t E [0, T] : f e^\Y s \ 2 dA s + f f e^\Z s (y)\ 2 Mdy)dA s > n}, 
Jo Jo Jk 

with the convention inf = T. Computing the Ito differential d{eP At \ Yt\ 2 ) on the interval [0, S n ] 
and proceeding as before we deduce 

(3E e^\Y s \ 2 dA s +E eP A °\Z s {y)\ 2 <t> s {dy)dA s 
Jo Jo „ Jk 



< E e? As ™ \Y Sn \ z + 2E / ef A ' Y s f s dA s . 
Jo 

Using the inequalities 2Y s f s < {J3/2)\Y S \ 2 + (2//3)|/ s | 2 and (^T2l) (with S = S n ) we find the 
following estimates 



E 



E 



S ,i 



10 JK 

from which we deduce 
E 



I"" e^\Y s \ 2 dA s < lEe^|e| 2 + ^E f \ f s \ 2 dA s , 
Jo P P Jo 



e^\Z s (y)?Udy)dA s < 4Ee^|£| 2 + ^E / e^\f s \ 2 dA s 



Y s \ 2 dA s +E 



JK 



^\Z s (y)\ 2 ( j )s (dy)dA s 



< Cl (/3)Ee^| 2 + c 2 (/3) [ T e^\f s \ 2 dA 

Jo 



(3.13) 



where ci(/3) = 4(1 + ±) and c 2 (/3) = |(1 + £). 
Setting S 1 = lim n S n we deduce 



[ S e^\Y s \ 2 dA s + f [ e^\Z s (y)\ 2 Mdy)dA s 
Jo Jo Jk 



< oo, 



a.s. 



which implies S = T, F-a.s., by the definition of S n . Letting n — > oo in (|3.13p we conclude that 
holds, so that (Y, Z) E K^. □ 
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Theorem 3.4 Suppose that Hypothesis \3. 1\ holds with > L 2 + 2L' . 

Then there exists a unique pair (Y, Z) in KP which solves the BSDE ([33 



Proof. We use a fixed point theorem for the mapping V : ~KP — > K.P defined setting (Y, Z) = 
T(U, V) if (Y, Z) is the pair satisfying 

Y t + [ T [ Z s (y) q(ds dy) = £ + / f s (U s , V s ) dA s . (3.14) 
Jt Jk Jt 

Let us remark that from the assumptions on / it follows that E J Q T e^ Aa \f s (U s , V s )\ 2 dA s < oo, so 
by Lemma 13.31 there exists a unique (Y, Z) E satisfying (|3.14j) and T is a well defined map. 

Let (U i ,V i ),i = l, 2, be elements of and let {Y\ Z l ) = F(U\ V { ). Denote Y = Y 1 -Y 2 , 
~Z_ = Z 1 - Z 2 , U = U 1 - U 2 , V = V x -V 2 , J s = f s (Ul,V}) - f s {U 2 , V 2 ). Lemma HO] applies to 
Y, Z, f and (|3.7|) yields, noting that Yt = 0, 



Ee^\Y t \ 2 + (3E F e? A °\Y s \ 2 dA s + E P [ e^\Z s (y)\ 2 Mdy)dA s 
Jt Jt Jk 

= 2E^ e^YJ s dA s , t 6 [0, T], 
From the Lipschitz conditions of / and elementary inequalities it follows that 

cT f-T 



(3E [ e^\Y s \ 2 dA s + E f [ e^\Z s (y)\ 2 Mdy)dA s 
Jo Jo Jk 

< 2LE j t e pAa \Y s \(j^\V s {y)\ 2 4> s {dy)\ dA s + 2L'E J e^|Y s | \U.\ dA s 

<aE [ [e^\V s {y)\ 2 cp s {dy)dA s + —E[ e? A '\Y a \ 2 dA, 
Jo Jk^ a Jo 

+ 7 L'E f e^ As \Y s \ 2 dA s + — E / e pA " \U S \ 2 dA s 
Jo 7 Jo 



for every a > 0, 7 > 0. This can be written 

/5-^-7i') \Y\}+\\w?<cx\\n}+^m 

By the assumption on (3 it is possible to find a G (0, 1) such that 

a \ a 



If L' = we see that T is an a-contraction on endowed with the equivalent norm (Y, Z) 1— > 
(/? - (L 2 /a)) \Y\l + If L' > we choose 7 = l/i/a and obtain 

-j= \Y\ 2 + ||Z||| < a\\V\\ 2 + L'^\U\ 2 = a \U\ 2 + ||T/|| 2 ) , 

so that r is an a-contraction on K' 3 endowed with the equivalent norm (Y, Z) i— > (L' / y/a) |Y|o + 
\\Z\\ 2 . In all cases there exists a unique fixed point which is the required unique solution to the 
BSDE (EH). □ 



We next prove some estimates on the solutions of the BSDE, which show in particular the 
continuous dependence upon the data. Let us consider two solutions (Y-^Z 1 ), (Y 2 ,Z 2 ) £ K@ 
to the BSDE (|3.ip associated with the drivers f 1 and f 2 and final data £ x and £ 2 , respectively, 
which are assumed to satisfy Hypothesis 13.11 Denote Y = Y 1 — Y 2 , Z = Z 1 — Z 2 , £ = £ — £ 2 , 
J s = f}(Y 2 ,Z 2 (.))-f 2 (Y 2 ,Z 2 (.)). 
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Proposition 3.5 Let (Y,Z) be the processes defined above. Then, for (3 > 2L' + L 2 , the a 

priori estimates hold: 

n * j^r^ AT ^ + Q _ w - l 2 ) 2 E f ^ dA - 

wm < { 2+ p -2L>-Li ) Ee " AT ® 2 



T 



f3-2L' -L 2 \ (3-2U -L 2 / 
Proof. From the Ito formula applied to e^ At \Y t \ 2 it follows that 

d{e pAt \Y t \ 2 ) = pe pAt \Y t \ 2 dA t + 2e /9 ^F t _dy t + e^ At \AY t \ 2 . 
So integrating on [t, T] and recalling that A is continuous, 

, t7 i2 



e^\Y t \ 2 = -/ Pe^\Y s \ 2 dA s -2 [ eP A >Y B _ [ Z s (y)q(ds dy) - V e^\AY s 
Jt Jt Jk t<s < T 

+e MT|||2 + 2 j\^s YsU \Y},Zl{.))- f 2 (Y 2 ,Z 2 (-))dA s . (3.15) 

The integral process J* * e /3 " 4s y s _ J - ^ Z s {y)q{ds dy) is a martingale, because the integrand process 
e P A "Y s -Z s (y) is in L 1 (p): in fact from the Young inequality we get 

E [ T [ e^\Y s _\\Z s (y)\Mdy)dA s 
Jo Jk 



io JK 

(■T 



<]-E ! e^\Y s .\ 2 dA s + ]-E [ [ e^\Z s (y)\ 2 Mdy)dA s < +oo. 
1 Jo 1 Jo Jk 



Moreover we have 

£ e^|AF s | 2 = f I e^\Z s (y)\ 2 p(dsdy) 



0<s<t 



JK 



f [ e^\Z s (y)\ 2 q(dsdy)+ f [ e^\Z s (y)\ 2 '<l> a {dy)dA 8 , 
Jo Jk Jo jk 



where the stochastic integral with respect to q is a martingale. 

Hence taking the expectation in 15 j) . by the Lipschitz property of the driver f 1 and using 
the notation ||z(-)||s = f K \z(y)\ 2 (j) s (dy) we get 

Ee^\Y t \ 2 = -E [ T /3e^\Y s \ 2 dA s -E F [ e^\Z s (y)\ 2 Mdy)dA s + Ee^|£| 2 
Jt Jt Jk 



+2E J e^Y s {f\Y}, Z\) - f 2 (Y 2 , Z 2 )dA s 

< _ E [ T (3e^\Y s \ 2 dA s -E F [ e^\Z s (y)\ 2 Mdy)dA s + Ee^|£| 2 

Jt Jt Jk 

+2E j\e A °\Y s \(\f\Y},Zl) - f\Y 2 ,Z 2 )\ + \J s \)dA s 

rT _ pT _ 

< -E !3e pAs \Y s \ 2 dA s -E \ e pAs \\Z s \\ 2 s dA s + Ee^ T |f| 2 

Jt Jt 

pT pT _ _ pT _ _ 

+2L'E / e^\Y s \ 2 dA s +2LE / e^\Y s \\\Z s \\ s dA s +2E / e pAa \Y a \\f a \dA a 
Jt Jt Jt 
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We note that the quantity Q(y, z) = — j3\y\ 2 — \\z\\ 2 s + 2L'\y\ 2 + 2L|y|||z|| s + 2|/ s ||y| which occurs 
in the integrand terms in the right hand of the above inequality can be written as 



Q(y,z) = -(3\y\' + 2L'\y\^+L 2 \y\ 2 + 2\f s \\y\-(\\4s-L\y\ 
= -fa{\y\ - /3Z 1 |Ll) 2 " (Ms ~ L\y\) 2 + fa^J s \ 2 

where fa := /3 — 2L' — L 2 is assumed to be strictly positive. Hence 



Ee^\Y t \ 2 + p L E [ T e^(\Y s \-P2 l \f s \) 2 dA s + E F e? A °(\\Z s \\ s - L\Y s \fdA a 
Jt Jt 

< Ee pAr |£| 2 + E [ T e? A ° 3^-dA s 

Jt PL 



from which we deduce 

r-T 



and 



e[ e^\Y s \ 2 dA s < A Ee ^T|||2 + 4 E f e p A ^- f ^ dA 
Jo fa pi Jo 



E [ T e^\\Z s \\ 2 dA s < f 2 + ^W^|£| 2 + J-fl + -^)E F e^-^dA 



fa) fa V fa) Jo fa 



s 



□ 



Prom the a priori estimates one can deduce the continuous dependence of the solution upon 
the data. 

Proposition 3.6 Suppose that Hypothesis I3.il holds with (3 > L 2 + 2L' and let (Y, Z) be the 
unique solution in to the BSDE / TO]) . Then 

E j T e^\Y s \ 2 dA s + E F eP*' f \Z s (y)\ 2 Mdy)dA s 
Jo Jo Jk 

< d(/3)Ee^|e| 2 + C 2 (/3) [ e ^SfidA s , 

Jo PL 

where C x {fl = (2 + j^u) > = J^U (l + TI=$=I?) ■ 

Proof. The thesis follows from Proposition 13.51 setting f 1 = f, ^ = f, f 2 = and £ 2 = 0. □ 



4 Optimal control 

Throughout this section we assume that a marked point process is given, satisfying the assump- 
tions of Section [2 In particular we suppose that T n — > 00 P-a.s. and that (|2.ip holds. 

The data specifying the optimal control problem are an action (or decision) space U, a 
running cost function I, a terminal cost function g, and another function r specifying the effect 
of the control process. They are assumed to satisfy the following conditions. 

Hypothesis 4.1 1. (U,U) is a measurable space. 

2. The functions r, I : £1 x [0, T] x K x U — >• R are V ® K, (g> U -measurable and there exist 
constants C r > 1, C\ > suc/i i/mi, P-a.s. , 

< r t (y,u) < C r , \l t (x,u)\ < C h t e [0,T],x e K,u e U. (4.1) 
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3. The function g : £1 x K — > R is J-y <8) IC-measurable. 

We define as an admissible control process, or simply a control, any predictable process 
(ut)t£\o,T] with values in U. The set of admissible control processes is denoted A. 

To every control u(-) G „4 we associate a probability measure ¥ u on (SI, J 1 ") by a change of 
measure of Girsanov type, as we now describe. We define 

L t = exp( [ I (1 - r s (y,u s )) (p s (dy)dA s S ) J [ r Tn (£ n ,u T J, t <= [0,T], 
^0 ^ / n>l:T„<< 

with the convention that the last product equals 1 if there are no indices n > 1 satisfying 
T n < t (similar conventions will be adopted later without further mention). It is a well-known 
result that L is a nonnegative supermartingale, (see |17| Proposition 4.3, or [3]), solution to the 
equation 

L t = l+ I [ L s ^(r s (y,u s )-l)q(dsdy), t € [Q,T\. 
Jo Jk 

The following result collects some properties of the process L that we need later. 



Lemma 4.2 Let 7 > 1 and 



2 



/3 = 7 + l + ^-. (4.2) 
7-1 

If Eexp(/3^4T) < 00 then we have sup tg [ ,T] ^-^7 < 00 anc ^ ELy = 1. 

Proof. We follow [6j, Chapter VIII Theorem Til, with some modifications. To shorten notation 
we define p s (y) = r s (y,u s ) and we denote L t = £{p)t- For 7 > 1 we define 

a s(y) = T^H 1 - PsivV 2 ), b s (y) = 7 - jp s (y) - 7" 1 + 7" 1 ^(y) 7 ' , 
so that 7(1 - p s (y)) = a s (y) + b s (y). Then 

L 7 = expQ^ J^a s (y) + b s {y))Udy)dA)j ]J p Tn {tn)\ 



and by Holder's inequality 



el 7 < t 



exp \L j K ias ^ y ^^ dy ^ dA ^) n pT "^ 

Eexpf/ / -^L-b s (y)<f> s (dy)dA 



n 2 



T n <t 

T-l 



Noting that ^a s {y) = 1 — p s (y) 7 , the term in square brackets equals £(/0 7 )t and we have 
E£(p 72 )t < 1 by the supermartingale property. Since 6 s (y) < 7 — 7 _1 + 7 _1 C 7 we arrive at 



2 7-1 

EL 7 < jEexp ^ r ^ 7 + l + -^_^| = {Eexp^r)} 2 ^ < 00. (4.3) 

Let S n = inf{i G [0, T] : + At > n} with the convention inf = T, and let pi (y) = 
l [0iSn] ( S )p s (y) + l ( s n ,T](s), ^ (n) = ^(/> (n) ). Then satisfies 

4 re) = 1 + ft l£> (4 n \y) - 1) q(dsdy), t G [0,T]. 
Jo Jk 
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By the choice of we have = L t /\s n , and by the choice of S n it is easily proved that 
E Jq J k L^2 \rt\y) - 1| <t> s {dy) dA s < oo, so that Z>) is a martingale and EL^ ) = EL tASn = 
1. The first part of the proof applies to L^ 1 and the inequality (|4.3|) yields in particular 
sup n E(L| n ' ) ) 7 = sup n E(L iA 5 n ) 7 < oo. So (L t /\s„)n is uniformly integrable and letting n — >■ oo 
we conclude that ELj = 1. □ 

Under the assumptions of the lemma, the process L is a martingale and we can define a 
probability P„ setting ¥ u (dw) = LT{oj)f{dui). It can then be proved (see [IT] Theorem 4.5) that 
the compensator p u of p under ¥ u is related to the compensator p of p under P by the formula 

p u (dtdy) = r t (y,u t )p(dtdy) = r t (y,u t ) (f> t (dy) dA t . 

In particular the compensator of N under P u is 

A t = f I r s (y,u s )Mdy)dA s . (4.4) 
Jo Jk 

We finally define the cost associated to every u(-) £ A as 



J(u(-))=E U 



[ T l t (X t ,u t )dA t + g(X T ) 
Jo 

where E u denotes the expectation under F u . Later we will assume that 

E[\g(X T )\ 2 e^]<oo (4.5) 

for some ft > that will be fixed in such a way that the cost is finite for every admissible control. 
The control problem consists in minimizing J(u(-)) over A. 

Remark 4.3 We recall (see e.g. [6], Appendix A2, Theorem T34) that a process u is (J-t)- 
predictable if and only if it admits the representation 

u(u,t) =^« (n) (w,t) 1t„(u0<*<t„+i(") ( 46 ) 

n>0 

where for each n > the mapping (oj,t) h-> ii( ra )(o;,t) is Ft u ® S([0, oo))-measurable. Since we 
have Tt u = a(Ti,^i,0 < i < n) (see e.g. jS], Appendix A2, Theorem T30) the fact that a control 
is predictable can be roughly interpreted by saying that the controller, at each time T n , based 
on observation of the random variables Tj,£j,0 < i < n, chooses his present and future control 
actions and updates his decisions only at time T n+ \. 

Remark 4.4 We notice that the laws of the random coefficients r, l,g under P and under F u 
are not the same in general, so that the formulation of the optimal control problem should 
be carefully examined when facing a specific application or modeling situation. This difficulty 
clearly disappears when r, l,g are deterministic. 

We next proceed to the solution of the optimal control problem formulated above. A basic 
role is played by the BSDE 

Y t + f [ Z s {y)q(dsdy)=g{X T )+ f f(s,X s ,Z s (-))dA s , t e [0,T], (4.7) 
Jt Jk Jt 
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with terminal condition g{Xx) and generator denned by means of the hamiltonian function /. 
The hamiltonian function is defined for every oj G t G [0, T],x G K and z G C l {K, fC, 0t(w, dy)) 
by the formula 

f(w,t,x,z(-)) = inf \l t (u,x,u) + / z(y) (r t (u,y,u) - 1) (f> t (uj , dy) \. (4.8) 

We will assume that the infimum is in fact achieved, possibily at many points. Moreover we 
need to verify that the generator of the BSDE satisfies the conditions required in the previous 
section. It turns out that an appropriate assumption is the following one, since we will see below 
(compare Proposition 14. 8p that it can be verified under quite general conditions. Here and in 
the following we set Xq- = Xq. 

Hypothesis 4.5 For every Z G C 1,0 (p) there exists a function u z : f2 x [0,T] — > U, measurable 
with respect to V and IA, such that 

f(u,t,X t -(u),Z t (u),-)) = l t (X t ^(uj),u z (uj,t)) + J Z t (u,y) (r t (u , y , u z (u , t)) - l) <p t (uj,dy) 

(4-9) 

for almost all (oj,t) with respect to the measure dAt(uj)¥(duj). 

Note that if Z G £ 1,0 (p) then Z t (uj,-) belongs to C l (K,K.,4>t{oJ-,dy)) except possibly on a pre- 
dictable set of points (u),t) of measure zero with respect to dAt(oj)¥(duj), so that the equality 
(|4.9p is meaningful. Also note that each u z is an admissible control. 

We can now verify that all the assumptions of Hypothesis 13.11 hold true for the generator of 
the BSDE (|4,7p . which is given by the formula 

f t (u,z(-)) = f(u,t,X t (to),z(-)), u;€n,t€ [0,T], z G C 2 (K, JC, dy)). 

Indeed, if Z G C 2 ^(p) then Z G C lfi (p) by (^3]l . and shows that the process (u,t) ^ 

f(uj, t, X t -(uj), Zf(u), ■)) is progressive; since A is assumed to have continuous trajectories and X 
has piecewise constant pahts, the progressive set {(uj,t) : Xt-(uj) ^ X t {ui)} has measure zero 
with respect to dAt(u))F(doj); it follows that the process 

(uj,t) h- f(u,t,X t (u)),Z t (u},-)) = f t (u),Z t (u),-)) 

is progressive, after modification on a set of measure zero, as required in ()3.2p . Next, using the 
boundedness assumptions ([4.ip . it is easy to check that (|3.3p is verified with V = and 

L = sup \r — 1| = sup{|rt(y, u) — 1| : oj G O, t G [0,T], y G K, u G f/}. 

Using (|4.ip again we also have 

E [ T e 0A *\f(t,X t ,O)\ 2 dA t = E /^e^l inf / t (X t , u)] 2 ^ < Cf^Ee^, (4.10) 

so that ()3.4p holds as well provided the right-hand side of (|4,10p is finite. Assuming finally that 
(|43]) holds, by Theorem E31 the BSDE has a unique solution (Y, Z) G IfC' 3 if f3 > L 2 . 

The corresponding admissible control u , whose existence is required in Hypothesis 14.51 will 
be denoted u*. 

We are now ready to state the main result of this section. Recall that C r > 1 was introduced 
in (IP) . 
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Theorem 4.6 Assume that Hypotheses \4-l\ and \4.5\ are satisfied and that 

oo. 

Suppose also that there exists j3 such that 

P > sup|r- 1| 2 , Eexp(/L4 T ) < oo, E[\g(X T )\ 2 e^ AT ] < oo. 



(4.11) 



(4.12) 



Let (Y, Z) £ K/ 3 denote the solution to the BSDE (|47f|) and u* = u the corresponding admissible 
control. Then u*(-) is optimal and Yq is the optimal cost, i.e. Yq = J(u*(-)) = inf^.)^ «/(«(•)). 

Remark 4.7 Note that if g is bounded then (|4.12p follows from (|4.1ip with /3 = 3 + C 4 , since 
\r t (y,u)-l\ 2 < (C r + 1) 2 <3 + C r 4 . 

Proof. Fix u(-) G .4. Assumption (|4,lip allows to apply Lemma 14.21 with 7 = 2 and yields 
EL 2 - < 00. It follows that g(Xx) is integrable under P n . Indeed by (|4.5p 



E U | 5 (X T )| =E|L T5 (X T )| < (El£) 1/2 (%(*r) 



2U/2 



< 00. 



We next show that under P u we have Z € £ 1,0 (p), i.e. E u f Q / x \Z t (y)\p u (dtdy) < 00. First 
note that, by Holder's inequality, 



jk 



\Z t (y)\Mdy)dA t 



e-i M ei M \Z t {y)\4> t {dy)dA t 



< 



JK 
T 



e~ pAt dA t 



1/2 



1 - e" 



0At \ 1/2 / r T 







1/2 



/3 



Jk 



e? At \Z t (y)\ 2 Mdy)dA t 



1/2 



Therefore, using (|4.ip . 



E u 



Jft: 



|Z t (y)|p»(cftdy) 



E u 
E 



\Zt{y)\r t (y,u t )4>t{dy)dA t 



\Z t (y)\r t (y,u t )(j>t(dy)dA t 



Jft: 



(y)\ 2 Mdy)dA t 



1/2 



and the right-hand side of the last inequality is finite, since (Y,Z) el' 5 . We have now proved 
that Z € C l '°(p) under P u . 
In particular it follows that 



E„ 



Jk 



Z t (y)p(dtdy) = E u 



T 



JK 



Z t {y)p u {dtdy)=E u 



Jk 



Ztiy) r t {y,u t )4>t{dy)dA t 



Setting t = and taking the expectation E u in the BSDE (|4.7p . recalling that q(dtdy) 
p(dtdy) — p{dt dy) = p{dt dy) — 4>t{dy) dA t and that Yq is deterministic, we obtain 

rp rp 

Y + E u [ [ Z t (y)(r t (y,u t )-l)Mdy)dA t = E u g(X T )+E u [ f(t,X t ,Z t (-))dA t . 

JO JK JO 
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We finally obtain 
Y = J(u(-))+E u 

= J(n(-))+E» 



T 



T 



f(t,X t ,Z t (-))-l t (X t ,u t )- 
f(t,X t .,Z t (-))-k(X t .,u t ) 



K 



Zt(y) (n(y,u t ) - l)M d v) 



dA t 



Zt(y) (r t (y,u t ) 



1) 



t{dy) 



K 



dA t , 



where the last equality follows from the continuity if A. This identity is sometimes called the 
fundamental relation. By the definition of the hamiltonian /, the term in square brackets is 
smaller or equal to 0, and it equals if u(-) = u*(-). □ 
Hypothesis 14.51 can be verified in specific situations when it is possible to compute explicitly 
the functions u z . General conditions for its validity can also be formulated using appropriate 
selection theorems, as in the following proposition. 



Proposition 4.8 In addition to the assumptions in Hypothesis ]^. 1\ suppose that U is a compact 
metric space with its Borel a-algebra U and that the functions r t (u), x, •), Zt(w, x, •) : U — > M are 
continuous for every uj G Q, t G [0,T], x G K. Then Hypothesis \4- 5\ is verified. 



Proof. Let us consider the measure n(du) dt) = dA t (uj)F(du}) on the predictable cr-algebra V . 
Let V denote its //-completion and consider the complete measure space (O x [0, T],V,/J,). Fix 
Z G £ 1,0 (p), note that the set A z = {{uj,t) : Z t (u:,-) ^ C}{K, /C, 4> t (to, dy)) has //-measure zero 
and define a map F z : Q x [0, T) x U -> E setting 



F z (uj,t,u) 



lt(w,X t -(w),u) + J Z t (ui,y)(r t (oJ,y,u)-lj(t)t(ui,dy) if (w,t) $ A z , 
if(uj,t)eA z . 



Then F z (-, ■,u) is P-measurable for every u G U, and it is easily verified that F z (u,t,-) is 
continuous for every (u),t) G fix [0, T]. By a classical selection theorem (see pQ, Theorems 8.1.3 
and 8.2.11) there exists a function u z : fi X [0, T] — > U, measurable with respect to V and Li, 
such that F z (u, t,u z (u}, t)) = min ueC / F z (oo, t, u) for every (u, t) G £1 x [0, T], so that P~9j) holds 
true for every (w, t). After modification on a set of /x-measure zero, the function u z can be made 
measurable with respect to V and U, and (|4.9p still holds, as it is understood as an equality for 
//-almost all (u,t). □ 

In several contexts, for instance in order to apply dynamic programming arguments, it is 
useful to introduce a family of control problems parametrized by (t, x) G [0, T] x K. Recall the 
definition of the processes (X t s ' x ) s£ ^ t ^ in subsection 12.41 

For fixed (t, x) the cost corresponding to u G A is defined as the random variable 



J t {x,u(-))=E: 



£ l s (Xl> x ,u s )dA s +g(X, 



where E^' denotes the conditional expectation under ¥ u given Tt- We also introduce the (ran- 
dom) value function 



v(t, x) = ess inf Jt(x, u(-)), t G [0, T], x G K. 
u(-)eA 

For every (t, x) G [0, T] x K we consider the BSDE 
Y^ x + [ T [ Z^(y) q (drdy)=g(X t /)+ f f(r,X^ x ,Z^(-))dA r , 
We need the following extended variant of Hypothesis 14.51 where we set X 



se[t,T]. (4.13) 



t,x 

t- 



x: 
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Hypothesis 4.9 For every (t,x) E [0, T] x K and every Z E £ 1,0 (p) £/tere exists a function 
yZ,t,x : ft x [t,T] —> U, measurable with respect to V and IA, such that 

f(u>, s, Xt'*(u), Z s (u>, •)) = l t {X s ^),u z ^ x {uj, s))+J Z s (uj, y) (r s (u, y,u z ^ x {uj, s))-l) </>.(u, dy) 

for almost all (oj,s) G Q x [t,T] with respect to the measure dA s (oj)¥(duj). 

This holds for instance if U is a compact metric space and the functions rt(u>, x, •), It (uj, x, •) : 
U — > R are continuous for every uj G O, t G [0,T], x E A\ 

In this situation Theorem 13,41 can still be applied to find a unique solution (Ys' x , Zs' x ) S £[t,T]- 
Let us now extend the process Z t,x setting Zl' x = for s E [0,t). The corresponding admissible 
control u Z ' t,x , whose existence is required in Hypothesis 14.91 will be denoted u* ,t,x (we set 



,*,t,x 



(oj,s) equal to an arbitrary constant element of U for s E [0, t)). 



Theorem 4.10 Assume that Hypotheses \4-l\ and \4-9\ are satisfied and that 

Eexp ((3 + C$)A T ) < oo. (4.14) 
Suppose also that there exists (5 such that 

/3 > sup|r- 1| 2 , Eexp(/L4 T ) < oo, E[| 5 (X^f)| V Ar ] < oo, t E [0, T], x E K, (4.15) 
(in particular, ()4.15|) follows from (|4.14p with ft = 3+C^ i/p zs bounded). For any (t,x) E [0,T]x 



A' Ze* (Y*>* Z*^) denote the solution of the BSDE gJ^j and u*'*' 1 
admissible control. 
Then u" 



,Z,t,x 



the corresponding 



is optimal and Y^' x is the optimal cost, i.e. Y{' x = Jt(x,u*(-)) = v(t,x) 



-a.s. 



The proof of Theorem 14.101 is entirely analogous to the proof of Theorem 14.61 the only 
difference being that in the BSDE one takes the conditional expectation E^ 7 ' instead of the 
expectation E u . 

Remark 4.11 1. Let u E A. Then, under P n , the compensator of the process N is A u 
defined in (|4.4p . It might therefore be more natural to define as the cost corresponding to 
u E A the functional 



[ T l t (X t ,u t )dA^+g(X T ) 
Jo 



l t (X t ,u t ) / r t (y,u t )4>t{dy)dA t + g{X T ) 
o JK 



instead of J(u(-)). This cost functional has the same form as J(u(-)), with the function 
I replaced by l®(x, u) := l t (x, u) f K r t (y, u) 4>t(dy). Since 1° is V <8> K, <8> W-measurable and 
bounded, the statements of Theorems 14.61 and 14.101 remain true without any change. 

2. Suppose that the cost functional has the form 



J 1 (u(-))=E l 



C ( T n,X Tn ,U Tn ) 

n>l:T„<T 



for some given function c : x [0, T] x K x U — > R which is assumed to be bounded and 
V <8> /C ® ^-measurable. It is well known (see e.g. [6], chapter VII, §1, remark (/?)) that we 
can reduce this control problem to the previous one noting that 



J\u(-))=M V 



K 



c(t,y,ut)p(dtdy) = E v 



K 



c(t, y, ut) r t (y, u t ) Mdy) dA t . 
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Thus, J 1 («(•)) has the same form as J(u(-)), with g = and the function / replaced by 
l\{x,u) := f K c(t,y,u)r t (y,u)(f)t(dy). Since I 1 is V £5 K <8> ^-measurable and bounded, 
Theorems 14.61 and 14.101 can still be applied. 

Similar considerations obviously hold for cost functionals of the form J(u(-)) + J 1 (it(-)). 



5 The stochastic Hamilton-Jacobi-Bellman equation 

Throughout this section we still assume that a marked point process is given, satisfying the 
assumptions of Section [2 In particular we suppose that T n — > oo P-a.s. and that (|2.ip holds. 

We address the same optimal control problem as in the previous section. The associated 
stochastic Hamilton-Jacobi-Bellman equation (HJB for short) is a backward stochastic differen- 
tial equation for unknown random fields on [0, T] x K, having the Hamiltonian function defined 
in (|4.8p as a nonlinear term. Before introducing the HJB equation we need a preliminary result 
which may have an interest in its own and will be used to clarify the connections with the opti- 
mal control problem and the BSDEs introduced in the previous section, as well as in the proof 
of the main result, Theorem 15.41 



5.1 A lemma of Ito type 

The Ito formula for processes defined by stochastic integrals with respect to random measures is 
certainly known, see e.g. [16J: it gives a canonical decomposition of v(t,Xt) for a deterministic 
functions v(t, x) smooth enough. We need an extension to the case when v(t, x) is stochastic and 
itself defined by integrals with respect to random measures. The following result is therefore the 
analogue to the so-called Ito-Kunita formula (also attributed to Bismut and Wentzell, see e.g. 

m, mm)- 

Lemma 5.1 Assume that v, f : ft x [0, T] x K — > R are Prog (g> K, -measurable, V : S7 x [0, T] x 
K x K — > M is V ® 1C ® fC- measurable, and, P-a.s. 



f f(s,x)dAs+ [ f V(s,x,y)q(dsdy), t € [0, T], x € K. (5.2) 
Jo Jo Jk 



[ \f(t,x)\dA t + I ! \V(t,x,y)\Mdy)dA t <oo, x e K. (5.1) 
jo Jo Jk 

Suppose that, P-a.s. 

v(t, x) — v(0, x) 
Then, P-a.s. 

v(t,X t )-v(0,X o ) = J f(s,X s )dA s + J J (y(s-,y) - + V(s,y,yj) p{dsdy) 

-[ [ V(s,X s ,y)<p s (dy)dA s , t € [0, T], x € K. 
Jo Jk 

(5.3) 



//, in addition, 



JK 



\v{t,y) + V{t,y,y)\<p t {dy)dA t < oo, P-a.s. 
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then, P-a.s. 
v(t,X t )-v(0,X o ) 



J f(s,X s )dA s + f^(v(s-,y)-v(s-,X s -) + V(s,y,y)) q(dsdy) 
(v(s,y) -v(s,X s ) +V(s,y,y) - V(s,X s ,y)j (p s (dy)dA s , 



t 

o Jk 



(5.4) 

for every t £ [0, T\, x € K. 

Remark 5.2 1. It follows from (|5.2p that P-a.s. the trajectories v(-,x) are cadlag for every 
x G K. Therefore the process (v(t—,x)) is well defined and V <8> /C-measurable. 

2. We note that 

pT p i-T n /\T p 

/ \V(t,X t ,y)\<t>t(dy)dA t = J2 \V{t,^ x ,y)\4>t{dy)dA t <^, P-a.s. 

JO JK n>\ Jin— i AT J K 



This follows from the assumption (|5.ip . and the fact that the sum is finite P-a.s. due to 
the assumption that T ri — > oo. Similarly, 

I \f(t,X t )\dA t + [ \v(t,X t )\dA t < oo, P-a.s. 

so that all the integrals above are well defined: compare the discussion in subsection [X 
Proof. Noting that there are Nt jump times T n in the time interval [0, t] we have 

N t 

v(t,X t )-v(0,X o ) = £ (v{T n -,X Tn )-v{T n „ x -,X Tn _S)+v{t,X t )-v{T N -,X TNt ), 

n=l 

where we use the convention v(0—,x) = v(0,x). Since X t = Xt n we have 

v{t,X t )-v(0,X o ) = I + II, 

where 

I = (v(T n -,X Tn ) - vF^Xt^)), 

n=l 

N t 

H = J2 (viTn-^T^-viT^-^T^+V&XTj-viTx-^T^). 
n=l 

Letting H denote the V <g> /C-measurable process 

Hs(y) = v(s-,y) - v(s-,X a -), 
with the convention Xq_ = Xq, we have 

1= Yl (v{T n -,X Tn )-v{T n -,X Tn _ 1 ))= Y, H Tn {X Tn )= f f H s {y)p{dsdy). 

n>l:T n <t n>V.T n <t J ° JK 

For n = 1, . . . , Nt, recalling that q(dt dy) = p(dt dy) — 4>t{dy) dA t and the definition of p, 
v(T n -,x) - u(T n _i-,x) 



pT n p pT n 
F(T n _i,x,^„_i) - / / V(s,x,y)(j) s (dy)dA s + / f{s,x)dA s 

JT n -\JK JTn-X 
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Setting x = Xx n _ t = £n-i> noting that X s = Xx n _ 1 for s £ (T n _i,T n ) and recalling that A is 
assumed to be continuous, 

v(T n -,X Tn ^) - v{T n ^ x -,X Tn _ x ) 

= y(T n _i,^_i,en-i)- / V(s,X s ,y)Mdy)dA s + f(s,X s )dA s . 

Similarly, 

v(t, X TNt ) - v(T Nt -,X TNt ) ^ 

= V(T Nt ,Z Nt ,t Nt )- f [ V(s,X s ,y)cf )s (dy)dA s + [ f(s,X s )dA s . 

J Tat. Jk Jt n , 



It follows that 



ft 

,y)<j) s (dy)dA s + I f{s,X s )dA s 



11= V nr„, f [ V(s,X s ,y)Mdy)dA s + f 
n>v. Tn <t Jo Jk Jo 

V(s,X s ,y)Mdy)dA s + [ f(s,X s )dA s , 

Jo 



/ V(s,y,y)p(dsdy) - 
o Jk Jo Jk 

and (|5.3p is proved. Using again the equality q(dt dy) = p(dt dy) — 4>t{dy) dA t and the additional 
assumption, (|5.4p follows as well. □ 



Remark 5.3 In differential form, under the assumptions of the lemma, if 

dv(t,x) = f(t,x)dA t + / V{t,x,y)q{dtdy) 

Jk 

then 

dv{t,X t ) = f(t,X t )dA t + J (y(t-,y)-v(t-,X t -) + V(t,y,y))q(dtdy) 
+ J (v(t, y) - v(t, X t ) + V(t, y, y) - V(t, X t , y)) &(dy) dA t . 

5.2 The equation 

In the rest of this section we will suppose that U,l,r,g are given satisfying Hypotheses 14.11 and 
14.51 as before. For technical reasons we will also assume that the space K is finite or countable 
(and /C is the collection of all its subsets) . We next present the HJB equation by first introducing 
the space of processes where we seek its solution. 

A pair (v, V) is said to belong to the space Wp, where (3 £ R, if 

1. v : O x [0,T] x K -> K is Prog <g> ^-measurable, V : O x [0, T) x K x K -> K \sV®K,®1C- 
measurable; 

2. The following is finite: 

:= supE [ T v(t,x) 2 e^dA t + E [ T v(t,X t ) 2 eP At dA t 

xeK Jo JO 
rT 



+ supl/ [ V(t,x,y) 2 Mdy)e^ At dA t 
x&k^ Jo Jk 

+E f [ \v(t,y) + V(t,y,y)\ 2 Mdy)e (3At dA t . 
Jo Jk 
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The space Hg, endowed with the norm ||| • |||^, is Banach space, provided we identify pairs of 
processes whose difference has norm zero. 

Let / be the Hamiltonian function defined in (|4.8p . A pair (v, V) G Big is called a solution 
to the stochastic HJB equation if, P-a.s., 



v(t,x) + 



T 



't JK 
g(x) + 



V{s, x, y) q(ds dy) 

^ M -.(.,,) + V(..v.v)-^..^,))*.(*)^ 
+ J f(s,x,v(s,-)-v(s,x) + V(s,-,-))dA s , te[0,T],xeK. 

We will also use the differential notation: 
-dv(t,x) = — 
+ 



(5.5) 



k v(T,x) 



V{t,x,y)q(dtdy) 

(y(t, y) - v{t, x) + V(t, y, y) - V(t, x, y) ) (p t {dy) dA t 

+fft, x, v{t, •) - v{t, x) + V(t, ■, •)) dA t , 
g{x), te[0,T],xeK. 



The basic result, which we assume for the moment and we will prove later, is the following. 
Let Po > 1 satisfy 

2(2L 2 + 3) | 8(2L 2 + 3) / | 1_ ; 



A) " 1 /?o 

Theorem 5.4 Lei -ftT 6e finite or countable and let Hypotheses \4-l\ and \4.9\ be verified. Suppose 
that there exists f3 such that 



/3>/3o, supE 



g(x) e 



< oo. 



(5.6) 



Then the HJB equation has a unique solution (v, V) in Wp. 
5.3 Application to control problems and BSDEs 

For every (t, x) G [0, T] x K we consider again the optimal control problem described just before 
Theorem 14. 101 and the BSDE ()4.13p for the unknown processes (Ys' x , Z t s ' x ) s£ y :tT y 

Let (v,V) G Wp the solution to the HJB equation constructed in Theorem 15.41 

Then we obtain the following result. 



Theorem 5.5 We make the same assumptions as in Theorem \5.4\ assuming in addition that (3 
also satifies (]4.15p . Then for every (t,x) £ [0,T] x K we have 



t,x 



v(s, Xr), Zr(y) = v(s-,y) - v(s-,X^) + V(s, y, y). 



(5.7) 



In particular, v(t,x) = Y^ ,x P-a.s. 

If (I4.14P also holds then v(t, x) coincides with the value function of the optimal control 
problem i.e. v(t,x) = ess inf u (.) g _4 Jt(x, u(-)) P-a.s. 
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Equalities (j5.7|) should be understood up to sets of measure zero in Q x [t,T], the measure 
being dA s (oj)¥(duj) for the first equality and (fi s (ui,dy)dA s (u})F(duj) for the second equality. 
Proof. We use a straightforward extension of the Ito lemma 15,11 to compute the stochastic 
differential dv(s, Xt' x ) on the interval [t,T] instead of [0,T]. Using the Lipschitz character of / 
it is not difficult to check that all the assumptions of the lemma are verified. For instance, we 
check that for every x G K 



E f T [ \V{t,x,y)\ Mdy)dA t < ( E f f \V(t,x,y)\ 2 Udy)e pAt dA t 

JO JK \ JO JK 



T 

S / e 

o 



■ 0At dA t 



is finite, since (v,V) G Wp and / Q T e~^ At dA t = /3 _1 (1 - e~^ r ) < so that V satisfies the 
required condition (|5.1[) . The other verifications are similar and are therefore omitted. 
The Ito lemma then yields 

v(s, Xl' x ) + f J (<r-,y) - v(r-,Xl'*) + V(r, y, y) q(dr dy) 



g(X t ' x )+ I f r (xp x ,v(r-,-)-v(r-,X^) + V(r r ,-))dA, s G [t,Tj. 



r- 



Comparing with equation (|4.13|) and setting 



Y. 



t.x 



v( S ,Xl' x ), %Y = v(s-,y) - v{s-X£) + V(s,y,y), 



we conclude that the pairs (Ys' x ,zl' x ) and (Ys' x , Zt' x ) are solutions to the same BSDE, and 
the latter also belongs to K^ 3 as it follows easily from the fact that (v,V) belongs to H«. By 
uniqueness for the solution to the BSDE, (|5.7p holds. 

All the other statements follow from Theorem 14.101 □ 



5.4 Proof of Theorem 15.41 

It is convenient to first state the following simple preliminary result. 
Lemma 5.6 Suppose 

-dv(t,x) = - V(t,x,y)q(dtdy) + U(t,x,y) <j> t {dy) dA t + u(t,x) dA t , v(T, x) = g(x) 
Jk Jk 



Then, setting cp = -gpj for (3 > 1, we have, for every x G K , 

e[ v(s,x) 2 e^ As dA s +E f [ V{s,x,yf(t) s {dy)e pAs dA s 
Jo Jo Jk 



< E 



g{xfe fiAT + C/3 E / u(s, xf e pAs dA s + C/3 E / / U(t, x, y) 2 <f> s (dy) e pAa dA s . 
- 1 Jo Jo Jk 



Proof. Using the identity (j3.7j) of Lemma 13.31 we have 
E v(t,x) 2 e? At +/3E / v(s,x) 2 e pAa dA 8 +K 





g(x) 2 e^ 






E 


+ 2eJ v(s,x) 













V(s,x,y) 2 Mdy)e (3As dA s 
e P A °dA*. 



>t JK 

U(t,x,y)(j) s (dy) + u(s,x) 
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Setting t = and using the elementary inequality 



2v(s, x) 



U(t,x,y) 4> s {dy) + u(s,x) 



K 



< ((3-l)v(s,x) 2 +c p 



U(t,x,y) 2 <p s (dy) + u(s,xf 



K 



the conclusion follows immediately. 

Proof of Theorem 15.41 We define a map r : 

H^, if (v, V) is the solution of 



□ 



setting (v, V) = T(u, U), for (u, U) G 



—dv(t,x) = —J V(t,x,y)q(dtdy) 

+ J (u(t, y) - u(t, x) + U(t, y, y) - V(t, x, y)) <j) t {dy) dA t 

+fft,x,u{t,-) -u(t,x) + U(t,-,-)\ dA t , 
{ v(T,x) = g(x), t g [0,T], xeK. 

Note the two occurrences of V in the right-hand side. For fixed x £ K, the existence of processes 
v(-, x), V(-,x, ■) solution to this equation follows from an application of Theorem 13.41 Since K is 
assumed to be at most countable, the corresponding integral equation holds simultaneously for 
every i £ [0, T] and x G K, with the exception of a P-null set. The rest of the proof consists in 
showing that (v, V) G and that T is a contraction for sufficiently large j3. We limit ourselves 
to proving the contraction property, since the fact that (y, V) G can be verified by similar 
and simpler arguments. 

Let (u\ U l ) G M./3 for i = 1, 2 and let (v l , V 1 ) = Y{u\ IP). Define v = v 2 — v 1 , V = V 2 — V 1 , 
u = u 2 -u 1 , U = U 2 -U 1 , 



f(t, x) = f(t,x, u 2 (t, •) - u 2 (t, x) + U 2 (t, ;-))-f[t, x, u 1 (t, •) - n 1 (t, x) + U 1 (t, ; •) 



Then 



-dv(t, x) 



+ 



k v(T,x) 



IK 



V(t,x,y)q(dtdy) 
u(t, y) - u(t, x) + U (t, y, y) - V(t, x, y) ) (p t (dy) dA t + f(t, x) dA t , 



0, t G [0,T], x € K. 



From Lemma 15.61 it follows that, for every x G K , j3 > 1 



(5.8) 



E / u(s,x)V As dA s + E 



+ 



0-1 



-E 



JK 



V(s,x,y) 2 Mdy)e^dA s < - 

JK P — 1 JO 

2 



T E f /( 



s,x) e 



2 ^dA, 



[u{s,y) - u(s,x) + U(s,y,y) - V{s,x,y)] <j) s (dy) e pAa dA s 



By the Lipschitz condition on / we have 



f(s, x) 2 < L 2 [u(s, y) - u(s, x) + U(s, y, y)] 2 4> s (dy), 



(5.9) 



K 



and it follows that, for every x G K , /3 > 1 



E / v(s,x) 2 e pAs dA s + E 



< 



2(2L 2 + 3) 



JK 

2 C PA S 



V(s,x,yYMdy)e pAs dA s 

-T 



E / u(s,xY e pAa dA s +E 



+ 



£-1 



-E 



r 



JK 



[u{s,y) + U(s,y,y)] Mdy)e l3As dA l 



o « 



V(s,x,y) 2 Mdy)e^dA 
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Setting cf := 2ly2 pJ[^ it follows that 



supE [ v(s,x) 2 e^dA s + supE [ [ V (s , x , y) 2 <j) s (dy) e? As dA s < c}\\\(u,U)\\\% (5.10) 
xeK Jo x£K Jo Jk 

We set now 

Y s = v(s,X s ), Z s (y) = v(s-,y) - v(s-,X s -) + V(s,y,y). 
Recalling (|5.8p and applying the Ito formula of Lemma 15.11 we obtain 

dY t = [ Z t (y)q(dtdy)-f(t,X t )dA t 

r 

+ / (z t (y)-u(t,y)+^t,X t )-U(t,y,y))<k(dy)dAt, 



and Yt = 0. Note that the term V(t, Xt,y) has disappeared. Using the estimate (|3.8p in Lemma 
3.31 on the BSDE we have 

E £ Y s 2 e PAs dA s +E £ j^Z s (y) 2 Mdy)e^dA s <Ul + Pj E £ f(s, X s ) 2 e? A *dA s 
+| (l + f) E J o J [Zs(y) - u(s,y) + u(s,X s ) - U(s,y,y)] 2 <j> s {dy)e^dA s . 



Using again the inequality (|5.9p we obtain 

E j' T Y 2 e^ As dA s + E [ T [ Z^yf^dy) e? A ° dA s 
Jo Jo Jk 

< 8(2 ^ +3) [l + -^E ! T u(s,X s ?e^dA s 



+ 8(2 ^ +3) (l + E f f [u(s, y) + U(s, y, y)] 2 Mdy) eP^dA. 
+ ^fl + ^jM £ J K Z s (y) 2 Mdy)e^dA s . 
Setting cf := 8(2L /j 2+3) (l + ±) it follows that 

E [ T Y 2 e^dA s +E f T [ ZM 2 Mdy)e^dA s < c}\\\(u,U)\\\ 2 - (5.11) 
Jo Jo Jk 

Recalling the definition of Y, Z and using the fact that A is assumed to be continuous we have 

E [ T [ [v(s,y) + V(s,y,y)] 2 <f> s {dy) e? A ° dA s = E f [ [Z s (y) + Y s ] 2 Mdy) e? A ° dA s 
Jo Jk^ Jo Jk 

<2E [ Y s 2 e^dA s + 2E / f Z s (y) 2 Mdy) eP^dA B < c 2 \\\(u, U)\\\ 2 
Jo Jo Jk 

(5.12) 

where the last inequality is due to (|5.1ip . Recalling that Y s = v(s,X s ), it follows from (|5.10j) . 
(HTTP . (f5TT2D that \\\(v, U)|||^ < cp \\\{u, U)\\\ 2 where cp = + c {2) is < 1 by the assumptions. 
This proves the required contraction property and finishes the proof. □ 
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